System and method for augmented reality detection of loose pharmacy items

ABSTRACT

A method includes capturing, by an image-capturing device, a one or more images of at least a portion of a pharmacy workstation. The method also includes identifying, by a processor in communication with the image capturing device, objects of interest in a first image of the one or more images and classifying, by the processor, the detected objects of interest using a convolutional neural network associated with the processor. The method also includes identifying, by the processor, a boundary defining an opening of a container in a second image of the one or more images. The method also includes updating, by the processor, an objects in container list based on a determination that at least one of the classified objects passed the boundary.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.16/779,223, which was filed Jan. 31, 2020. The entire disclosure of saidapplication is incorporated herein by reference.

TECHNICAL FIELD

This disclosure relates to pharmacy item detection and in particular, todetecting loose pharmacy items using augmented reality.

BACKGROUND

Pharmacists and pharmacy technicians typically receive requests tofulfill pharmacy orders that include various pharmacy items. Thepharmacy items may include medicine, medicine accessories, or othersuitable items. In order to fulfill the pharmacy order, pharmacytechnician (e.g., or the pharmacist) retrieves the items of a pharmacyorder from a stockroom and brings the items to a workstation where thepharmacy technician will fulfill the pharmacy order, which may includecounting quantities, generating labels, packaging items in a container,and the like.

Typically, such items in the stockroom are stored in bulk quality.Accordingly, when the pharmacy technician retrieves the items from thestockroom, the pharmacy technician typically determines how many of eachof the items are required to fulfill the pharmacy order. The pharmacytechnician may then update corresponding inventory lists in to indicatethe inventory quantities remaining after the pharmacy technicianfulfills the pharmacy order. In addition, after the pharmacy technicianprepares the items for the pharmacy order, the items in the pharmacyorder are typically verified, usually by a pharmacist. Such manualretrieval of stock items, inventory management, and verification can becost and resource intensive and may be prone to human error.

SUMMARY

This disclosure relates generally to pharmacy item detection.

An aspect of the disclosed embodiments is a method for fulfilling apharmacy order. The method includes capturing, by an image-capturingdevice, a one or more images of at least a portion of a pharmacyworkstation. The method also includes identifying, by a processor incommunication with the image capturing device, objects of interest in afirst image of the one or more images and classifying, by the processor,the detected objects of interest. The objects in the one or more imagesmay be classified using a convolutional neural network associated withthe processor. The method also includes identifying, by the processor, aboundary defining an opening of a container in a second image of the oneor more images. The method also includes updating, by the processor, anobjects in container list based on a determination that at least one ofthe classified objects passed the boundary.

Another aspect of the disclosed embodiments is an augmented realityapparatus for fulfilling a pharmacy order. The augmented realityapparatus includes an image-capturing device, a processor, and a displaydevice. The image-capturing device is configured to capture video datathat includes a one or more images of at least a portion of a pharmacyworkstation. The processor in communication with at least one memorythat includes instructions that, when executed by the processor, causethe processor to: read a first image of the one or more images of thevideo data; identify objects of interest in the first image; classifythe detected objects of interest (e.g., using a convolutional neuralnetwork); identify a boundary defining an opening of a container in asecond image of the one or more images; update an objects in containerlist based on a determination that at least one of the classifiedobjects passed the boundary; and generate information corresponding tothe pharmacy order based on, at least, the objects in container list.The display device is configured to display the information.

Another aspect of the disclosed embodiments is a system for fulfilling apharmacy order. The system includes a wearable augmented reality deviceand a machine learning device (e.g., convolutional neural network). Thewearable augmented reality device is configured to: capture video datathat includes a one or more images of at least a portion of a pharmacyworkstation; identify objects of interest in the one or more images; andidentify a boundary defining an opening of a container in the one ormore images. The machine learning device (e.g., a convolutional neuralnetwork) is configured to classify the objects of interest identified bythe wearable augmented reality device, wherein the wearable augmentedreality device identifies, in the one or more images, classified objectsof interest that pass the boundary of the container and provides, to awearer of the wearable augmented reality device, informationcorresponding to the pharmacy order in response to identifyingclassified objects of interest that pass the boundary of the container.

These and other aspects of the present disclosure are disclosed in thefollowing detailed description of the embodiments, the appended claims,and the accompanying figures.

BRIEF DESCRIPTION OF THE DRAWINGS

The disclosure is best understood from the following detaileddescription when read in conjunction with the accompanying drawings. Itis emphasized that, according to common practice, the various featuresof the drawings are not to-scale. On the contrary, the dimensions of thevarious features are arbitrarily expanded or reduced for clarity.

FIG. 1 generally illustrates a computing device according to theprinciples of the present disclosure.

FIG. 2 generally illustrates a loose pharmacy item detection systemaccording to the principles of the present disclosure.

FIG. 3 generally illustrates a guided access output according to theprinciples of the present disclosure.

FIG. 4 generally illustrates an order fill output according to theprinciples of the present disclosure.

FIG. 5 is a flow diagram generally illustrating a pharmacy order fillmethod according to the principles of the present disclosure.

FIG. 6 is a flow diagram generally illustrating an initial objectdetection model training method according to the principles of thepresent disclosure.

FIG. 7 is a flow diagram generally illustrating an object detectionmethod according to the principles of the present disclosure.

FIG. 8 is a flow diagram generally illustrating an object classificationdetection method according to the principles of the present disclosure.

FIG. 9 is a flow diagram generally illustrating a workstation restockingmethod according to the principles of the present disclosure.

FIG. 10 is a flow diagram generally illustrating an alternative pharmacyorder fill method according to the principles of the present disclosure.

DETAILED DESCRIPTION

The following discussion is directed to various embodiments of theinvention. Although one or more of these embodiments may be preferred,the embodiments disclosed should not be interpreted, or otherwise used,as limiting the scope of the disclosure, including the claims. Inaddition, one skilled in the art will understand that the followingdescription has broad application, and the discussion of any embodimentis meant only to be exemplary of that embodiment, and not intended tointimate that the scope of the disclosure, including the claims, islimited to that embodiment.

As described, pharmacists and pharmacy technicians typically receiverequests to fulfill pharmacy orders that include various pharmacy items.The pharmacy items may include medicine, medicine accessories, or othersuitable items. In order to fulfill the pharmacy order, pharmacytechnician (e.g., or the pharmacist) retrieves the items of a pharmacyorder from a stockroom and brings the items to a workstation where thepharmacy technician fulfills the pharmacy order. Such stockrooms may berelatively large and may include a plurality or shelves for storingvarious pharmacy items. Pharmacists and/or pharmacy technicians may haveto traverse multiple rows of shelves or access multiple rooms, which maybe a significant distance, in order to retrieve all required items forthe pharmacy order, which may be difficult and time consuming.

Additionally, items in the stockroom are typically stored in bulkquality. Accordingly, when the pharmacy technician retrieves the itemsfrom the stockroom, the pharmacy technician typically determines aquantity for each of the items required to fulfill the pharmacy order.In addition, after the pharmacy technician prepares the items for thepharmacy order, the items in the pharmacy order are typically verified,usually by a pharmacist. Such manual retrieval of stock items, inventorymanagement, and verification can be cost and resource intensive and maybe prone to human error.

Accordingly, systems and methods, such as those described herein, thatare configured to improve pharmacy order fulfillment by providingpharmacy order fulfilment automation and guided access to stockroomitems, may be desirable. In some embodiments, the systems and methodsdescribed herein may be configured to use augmented reality in order tofulfill a pharmacy order. The pharmacy order may be an order from apharmacy customer and may include pharmacy items, such as medicine(e.g., prescription or non-prescription), pharmacy accessories (e.g.,pads, syringes, measuring devices, and the like), other items, or acombination thereof.

The systems and methods described herein may be configured to useimage-capturing capabilities of an augmented reality device to capturevideo data that includes a plurality of images of at least a portion ofa pharmacy workstation. The systems and methods described herein may beconfigured to read or analyze the plurality of images of the video datato identify objects of interest in the video data. The objects ofinterest may include pharmacy items, such as medicine, pharmacyaccessories, and the like.

In some embodiments, the systems and methods described herein may beconfigured to use a trained machine learning network (e.g., a trainedconvolutional neural network) in communication with the augmentedreality device to classify the detected objects of interest. In someembodiments, the systems and methods described herein may be configuredto identify or read barcodes on at least some of the objects using themodel derived from the machine learning network (e.g., the convolutionalneural network) to classify the detected objects.

In some embodiments, the systems and methods described herein may beconfigured to update a workstation inventory list based on theclassification of the detected objects of interest. In some embodiments,the systems and methods described herein may be configured to identify aboundary defining an opening of a container in the video data. In someembodiments, systems and methods described herein are configured todetect the boundary defining the opening of the container using colorfiltering. In some embodiments, systems and methods described herein areconfigured to detect the boundary defining the opening of the containerusing edge detection. The edge detection may include Canny edgedetection or other suitable edge detection. In some embodiments, systemsand methods described herein are configured to detect the boundarydefining the opening of the container using line detection. The linedetection may include Hough transform line detection or other suitableline detection. Hough transformation line detection may include afeature extraction technique that uses a two-dimensional array (e.g.,referred to as an accumulator) to detect lines in images.

In some embodiments, the container may include a box, shippingcontainer, or other suitable container configured to enclose thepharmacy items of a pharmacy order. The container may include anopening, the boundary of which may be identified using the systems andmethods described herein. For example, the container may be an open toppolymer container, at which the systems and methods described herein maybe configured to can define the boundary (e.g., a virtual boundary) ofthe open top. In some embodiments, the systems and methods describedherein may be configured to update an objects in container list based ona determination that at least one of the classified objects passed theboundary, which may include the virtual boundary defined by the systemsand methods described herein. The systems and methods described hereinmay be configured to generate information corresponding to the pharmacyorder based on, at least, the objects in container list. In someembodiments, the systems and methods described herein may be configuredto display, using a display device, the information.

In some embodiments, the systems and methods described herein may beconfigured to update the workstation inventory list based on thedetermination that at least one of the classified objects passed theboundary. In some embodiments, the systems and methods described hereinmay be configured to provide, to the augmented reality device or adisplay of a computing device, an indication corresponding to theobjects in container list. In some embodiments, the systems and methodsdescribed herein may be configured to determine whether the objects incontainer list is the same as an expected objects list. In someembodiments, the systems and methods described herein may be configuredto display, on a display device, that the pharmacy order is fulfilledbased on a determination that objects in container list is the same asthe expected objects list.

In some embodiments, the systems and methods described herein may beconfigured to use a display of the augmented reality device to provideguided access to a user (e.g., a pharmacist and/or pharmacy technician)to a stockroom. For example, the augmented reality device may include adisplay system (e.g., eyeglasses or goggles) worn by the user. Thedisplay system may include a hands-free display. The systems and methodsdescribed herein may provide a map and/or navigation instructions to theuser by overlaying the map, the navigation instructions, and/or otherinformation on lenses of the display system, such that the userperceives the map, navigation instructions, and/or other informationwhile looking through the lenses at the real world (e.g., walkway,stockroom, and the like). The map, navigation instructions, and/or otherinformation may guide the user through the real world to stock itemsrequired for fulfilling the pharmacy order.

In some embodiments, the systems and methods described herein may beconfigured to train a trainable network (e.g., a convolutional neuralnetwork) to recognize various objects, such as pharmacy items, apharmacy workstation, containers associated with the pharmacy order,other suitable objects, or a combination thereof.

FIG. 1 generally illustrates a detailed view of a computing device 100according to the principles of the present disclosure. The computingdevice 100 may be configured to train a network (e.g., a convolutionalneural network) to recognize objects (e.g., such as those described), toprovide automation to pharmacy order fulfilment, and to provide guidesaccess to various pharmacy items in respective stockrooms or othersuitable locations. The computing device 100 may be any suitablecomputing device, such as a mobile computing device, a laptop computingdevice, a desktop computing device, a server-computing device, or anyother suitable computing device.

The computing device 100 may include a processor 102 configured tocontrol the overall operation of computing device 100. The processor 102may include any suitable processor, such as those described herein,which may be configured to load specific instructions and operate as adedicated device. The computing device 100 may also include a user inputdevice 104 that is configured to receive input from a user of thecomputing device 100 and to communicate signals representing the inputreceived from the user to the processor 102. For example, the user inputdevice 104 may include a button, keypad, dial, touch screen, audio inputinterface, visual/image capture input interface, input in the form ofsensor data, etc.

In some embodiments, the input device 104 may include or be incommunication with an augmented reality device 120. The augmentedreality (AR) device 120 may include any suitable augmented realitydevice configured to provide input, such as image data or other suitableinput, to the input device 104. In some embodiments, the augmentedreality device 120 includes a wearable augmented reality device, such aseyeglasses, goggles, or other suitable wearable augmented realitydevice. The wearable augmented reality device may include a transparentscreen that allows a user to see through the screen and allows images tobe displayed on the screen. The AR device 120 may include an inputmechanism, such as an image-capturing device (e.g., a camera or othersuitable image-capturing device), an audio-capturing device (e.g., amicrophone or other suitable audio-capturing device), one or moresuitable sensors (e.g., global position system sensors, gyroscopicsensor, accelerometer, other suitable sensors, or a combinationthereof), other suitable input mechanism, or a combination thereof.

The computing device 100 may include a display 106 that may becontrolled by the processor 102 to display information to the user. Adata bus 108 may be configured to facilitate data transfer between, atleast, a storage device 110 and the processor 102. The computing device100 may also include a network interface 112 configured to couple orconnect the computing device 100 to various other computing devices ornetwork devices via a network connection, such as a wired or wirelessconnection. In some embodiments, the network interface 112 includes awireless transceiver.

The storage device 110 may comprise a single disk or a plurality ofdisks (e.g., hard drives), and includes a storage management module thatmanages one or more partitions within the storage device 110. In someembodiments, storage device 110 may include flash memory, semiconductor(solid state) memory or the like. The computing device 100 may alsoinclude a memory 114. The memory 114 may include Random Access Memory(RAM), a Read-Only Memory (ROM), or a combination thereof. The memory114 may store programs, utilities, or processes to be executed in by theprocessor 102. The memory 114 may provide volatile data storage, andstores instructions related to the operation of the computing device100.

FIG. 2 generally illustrates a loose pharmacy item detection system 200according to the principles of the present disclosure. The system 200may be partially embodied in the computing device 100 or in anothercomputing device or plurality of computing devices (e.g., such as aserver farm or cloud-based computing system) in communication with thecomputing device 100. For example, the processor 102 may executedinstructions stored in the memory 114 to perform at least some of thefunctions of the system 200. In some embodiments, a portion of thesystem 200 may disposed on the AR device 120. For example, the AR device120 may include a corresponding processor that executes instructionsstored in a corresponding memory to perform at least some of thefunctions of the system 200.

The system 200 includes a robotic process automation (RPA) system 202.The RPA system 202 may be configured to interface and/or communicatewith an order website 204 to receive pharmacy orders. For example, apharmacy customer may access a customer interface disposed on the orderwebsite 204 or other suitable location. The customer may provide thecustomer interface with external order information 206. The externalorder information 206 may include prescription information indicating atype and amount of medicine and/or accessories. Additionally, oralternatively, the external order information 206 may includeinstructions indicating special conditions for the pharmacy order, suchas storage temperatures of certain medicine, and other suitableinformation. The RPA system 202 may receive a notification from theorder website 204 indicating a pharmacy order. Additionally, oralternatively, the RPA system 202 may access the order website 204 todetermine whether a pharmacy order is available.

The RPA system 202 may communicate the pharmacy order and anycorresponding information to process data 208. The RPA system 202 maystore the data as intermediate order fill data and/or other suitabledata. The process data 208 may include data or other information storedin a storage medium. For example, the process data 208 may be stored inthe storage device 110 of the computing device 100. In some embodiments,the process data 208 may be organized as a database or multipledatabases.

The system 200 includes an augmented reality (AR) hardware system 210.The AR hardware system 210 may be disposed on the computing device 100and/or on the AR device 120. For example, the AR device 120 may includea wearable device, such as goggles, or other suitable wearable device.Output from the AR hardware system 210 may be provided by the AR device120 and processing of the AR hardware system 210 may be disposed on thecomputing device 100. For example, the computing device 100 may receiveinput from the AR device 120 and may process the input to generateoutput that is displayed on a portion of the AR device 120, such as on alens or transparent display screen of the AR device 120. Alternatively,the AR hardware system 210 may be substantially disposed on the ARdevice 120.

In some embodiments, the AR hardware system 210 may be configured toprovide guided access to the user of the AR device 120. Guided accessmay include providing an augmented reality interface in order to providedirections to various pharmacy items in a corresponding stockroom. Theaugmented reality interface may include a map or instructions displayedon the lens, the transparent display screen, or other display device ofthe AR device 120.

As is generally illustrated in FIG. 3, the guided access may includedirectional symbols 302. The directional symbols 302 may indicate adirection the user is to take in order to traverse a path to variouspharmacy items associated with the pharmacy order or a location of oneor more pharmacy items on a shelf or other suitable location within thestockroom. The guided access may include item information 304. The iteminformation 304 may include a quantity of items that the user is toretrieve from the stockroom or other suitable information.

In some embodiments, the AR hardware system 210 may communicate with theprocess data 208 to retrieve data to generate the guided access. Forexample, the AR hardware system 210 may retrieve from the process data208 intermediate order fill data, workstation inventory data, othersuitable data, or a combination thereof. Additionally, or alternatively,the AR hardware system 210 may access warehouse data records 212 toretrieve stockroom layout data, inventory data, other suitable data, ora combination thereof

The warehouse data records 212 may be disposed on a remotely locatedcomputing device or computing system, such as a cloud computing system,a server farm, other suitable computing devices or computing system, ora combination thereof In some embodiments, the warehouse data records212 may be disposed on the AR hardware system 210. For example, ARhardware system 210 may download the warehouse data records 212 from aremotely locating computing device or computing server or the ARhardware system 210 or the AR hardware system 210 may be preprogrammedwith data from the warehouse data records 212.

In some embodiments, the AR hardware system 210 may generate data to bestored in the warehouse data records 212. For example, the AR hardwaresystem 210 may capture images of the stockroom accessed by the user ofthe AR device 120. As described, the AR device 120 may include animage-capturing device, such as a camera or other suitableimage-capturing device.

The image-capturing device may be configured to capture image datacorresponding to an environment within a field of view of the AR device120. The image data may include a plurality of images and/or video datacomprising a plurality of images of the environment within the field ofview of the AR device 120. The AR device 120 may include a neuralnetwork or may be in communication with a neural network remotelylocated from the AR device 120, such as a neural network on thecomputing device 100, a cloud computing device, a server in a serverfarm, or other suitable remotely located computing device. The neuralnetwork may include a convolutional neural network or other suitableneural network. As will be described, the neural network may be trainedto recognize aspects of the stockroom and/or inventory items in thestockroom. The AR device 120 may provide the image data to the neuralnetwork. The neural network may identify aspects of the stockroom and/orinventory in the stockroom. The neural network may then classify theidentified aspects of the stockroom and/or inventory in the stockroom inorder to generate data for the warehouse data records 212.

The AR hardware system 210 may generate the directional symbols 302, theitem information 304, other suitable symbols or information, or acombination thereof. As the user traverses the path to the inventoryitems required for the pharmacy order, the AR hardware system 210 mayhighlight inventory containers 306 on stockroom shelves or othersuitable location in the stockroom. The user may then perform a gestureto indicate that an inventory container 306 has been retrieved. Thegesture may include one or more motions to retrieve the inventorycontainer 306, a specific gesture, such as a predetermined handmovement, or other suitable gesture.

In some embodiments, the AR hardware system 210 may be configured torecognize the gesture and may provide a visual acknowledgement 308 thatthe user has retrieved the inventory container 306. The visualacknowledgement 308 may include a symbol, such as a checkmark, an emoji,or other suitable visual acknowledgement. The AR hardware system 210 maydisplay the visual acknowledgement 308, via one or more lenses of the ARdevice 120 or other suitable display. For example, the AR hardwaresystem 210 may overlay the visual acknowledgement 308 on the receivedinventory container 306.

In some embodiments, the AR hardware system 210 is configured to updatethe item information 304 in response to the user retrieving theinventory container 306. For example, the AR hardware system 210 maydetermine a remaining item count based on the contents of the inventorycontainer 306. The AR hardware system 210 may determine the contents ofthe inventory container 306 based on the inventory data stored in thewarehouse data records 212. The inventory data may indicate a type ofitems in the inventory container 306, a count of the items in theinventory container 306, other suitable data, or a combination thereof.The AR hardware system 210 may decrement a value associated with theitem information 304. For example, the AR hardware system 210 maygenerate item information 304 based on the intermediate order fill dataof the process data 208, the inventory data of the warehouse datarecords 212, other suitable data, or a combination thereof

In some embodiments, the system 200 may include a box fill detectionsystem 214. The box fill detection system 214 may be disposed on thecomputing device 100. In some embodiments, the box fill detection system214 may be disposed on the AR device 120, the computing device 100, aremotely located computing device, or a combination thereof. The boxfill detection system 214 includes an artificial intelligence 216 thatmay include a neural network, such as a convolutional neural network orother suitable neural network, a machine learning system, a computervision system, a deep learning system, other artificial intelligencesystems, or a combination thereof. For example, the artificialintelligence 214 may include or communicate with the convolutionalneural network described with respect to the AR hardware system 210.

The neural network of the artificial intelligence 214 may include, forexample, a deep learning network that typically includes various layersthat process inputs, such as the image data generated by the AR device120, and generate outputs (e.g., class scores, image classifications, orother suitable output). For example, the neural network can includeconvolution layers that process sets of inputs with convolution kernelsto generate sets of outputs. Convolutional layers are typicallyconfigured to detect high-level features of the inputs, such as edged,curves, simple colors, and the like.

The output of the convolutional layers may be provided to a fullyconnected layer. The fully connected layer typically connects everyneuron in one layer of the neural network to every other neuron inanother layer of the neural network. The fully connected layer isconfigured to receive inputs from the convolutional layers and generateoutputs that can be used to predict classifications for imagesassociated with the inputs. For example, during training of the neuralnetwork, as will be described, a plurality of images may be provided tothe neural network (e.g., using the convolutional layers, as described).The neural network may learn by using the fully connected layer toclassify each of the images.

The box fill detection system 214 may use the artificial intelligence216 to identify objects on a workstation 402 as is generally illustratedin FIG. 4. The box fill detection system 214 reads image frames of theimage data generated by the AR device 120. The box fill detection system214 may retrieve the image frames from accumulated training data 222.The accumulated training data 222 may communicate with a video framesegmentation and classification interface 224. The video framesegmentation and classification interface 224 may receive the image datafrom the AR device 120. The video frame segmentation and classificationinterface 224 may segment and classify the image frames of the imagedata. The accumulated training data 222 may store segmented andclassified image frames.

The box fill detection system 214, using the artificial intelligence216, may load an object detection model. The object detection model mayinclude a model generated during training of the artificial intelligence216. The box fill detection system 214 ma retrieve the model from theobject classification data 218. The artificial intelligence 216 reads afirst image frame of the image data and identifies objects in the firstimage frame. The box fill detection system 214, using the artificialintelligence 216, identifies the workstation 402 in the first imageframe.

The box fill detection system 214, using the artificial intelligence216, may identify various objects on the workstation 402 by reading thefirst image frame and/or one or more other image frames of the imagedata. For example, the box fill detection system 214 may identify asurface 404 and a display, such as the display 106 of the computingdevice 100, or other suitable display on the workstation 402. In someembodiments, the computing device 100 may output, via the display 106,pharmacy order information corresponding to the pharmacy order. Thecomputing device 100 may access the intermediate order fill data of theprocess data 208. The computing device 100 generates the pharmacy orderinformation using the intermediate order fill data.

The box fill detection system 214, using the artificial intelligence216, may identify a container 406 on the workstation 402. The userfilling the pharmacy order may package the pharmacy order in thecontainer 406. The container 406 may include a box, a prescriptionbottle, a bag, or any other suitable container.

The box fill detection system 214 may be configured to use theartificial intelligence 216 to identify a threshold 408 of the container406. The threshold 408 may include an edge or lip of the container 406.In some embodiments, the threshold 408 may define a plane perpendicularthe container 406. Additionally, or alternatively, the threshold 408 maydefine a boundary between items outside of the container 406 and itemsin the container 406.

In some embodiments, the box fill detection system 214 may be configureto identify items retrieved by the user filling the pharmacy order froma workstation inventory 412. For example, the user may retrieve apharmacy item 410 required for the pharmacy order from the workstationinventory 412. The box fill detection system 214, using the artificialintelligence 216, identify the item 410 using information on the item410. For example, the box fill detection system 214 may be configured toread a bar code on the item 410, a label on the item 410, a descriptionon the item 410, other suitable information on the item 410 or acombination thereof. The box fill detection system 214 may read an imageframe of the image data captured by the AR device 120.

The box fill detection system 214, using the artificial intelligence216, may first identify a general type of or class of item of the item410 by analyzing characteristics of the item 410, such as shape, color,size, other suitable characteristics, or a combination thereof. Forexample, the box fill detection system 214 may determine that the item410 is a container of medicine based on the shape, color, size, or acombination thereof of the item 410. The shape and size of the item maybe defined without regard to the orientation of the item such that theitem may be identified based on any orientation thereof. The box filldetection system 214 may then determine a specific type or class of theitem 410 using information 416 on the item 410.

For example, the box fill detection system 214 may attempt to identifythe item 410 by reading an identification mark on the item 410 (e.g., abarcode, an OR code, a two-dimensional (2D) data matrix symbol, analphanumerical identifier, or the like). For ease of description, thepresent specification may refer to bar codes for any of these uniqueidentifiers. If the box fill detection system 214 is unable to read oranalyze the information on the item 410 (e.g., because the informationis blocked or not facing the AR device 120), the box fill detectionsystem 214 may generate output to be provided to the AR device 120. Theoutput may include an indication, such as the indication 414,instructing the user of the AR device 120 to display information 416 onthe item 410 (e.g., by moving the item 410, such that the information iscaptured by the AR device 120). The box fill detection system 214 maygenerate a barcode reader box 420 (e.g., within a bounding box of theitem 410, as will be described).

The box fill detection system 214 may overlay the barcode reader box 420onto the image frame and/or one or more lenses of the AR device 120. Thebox fill detection system 214 may continue to read or analyze the item410 in the barcode reader box 420 until the box fill detection system214 is able to read the information 416 and identify the specific typeor class of the item 410. The box fill detection system 214 may comparethe information 218 to barcodes stored in the object classification data218 or the box fill detection system 214 may use a barcode decodermechanism to read the barcode. The box fill detection system 214determines the specific type of class of the item 410 based on theinformation 416 and a corresponding barcode on the object classificationdata 218. It should be understood that while only a barcode is describedherein, the box fill detection system 214 may use any suitableinformation to identify the specific type or class of the item 410.

The user may then place the item 410 in the container 406. The box filldetection system 214 may determine whether the item 410 has passed thethreshold 408 of the container 406. When the box fill detection system214 determines that the item 410 has passed the threshold 408 of thecontainer 406, the box fill detection system 214 determines that item410 has been placed in the container 406.

In some embodiments, the box fill detection system 214 may display, viathe AD device 210, order fill information 418. The order fillinformation 418 may include a count of items in the container 406, acount of items required to fill the pharmacy order, other suitableinformation, or a combination thereof. The box fill detection system 214may update the order information in response to determining that anitem, such as the item 410, has been placed in the container 406. Forexample, the box fill detection system 214 may decrement a count ofitems required to fill the pharmacy order and increment a count of itemsin the container 406. In some embodiments, the box fill detection system214 may generate output indicating that the pharmacy order is completein response to the count of items required to fill the pharmacy orderequals 0. The box fill detection system 214 may provide the output tothe AR device 120.

In some embodiments, the accumulated training data 222 may also includeorder fill videos with expected order data, warehouse inventory videoswith object data, other suitable data, or a combination thereof. Thedata stored in the accumulated training data 222 may be used to identifyobjects in the image data, as described. Additionally, or alternatively,the data in accumulated training data 222 may be used train or continueto train the system 200. For example, an interface to initiate modeltraining 220 may be used by the box fill detection system 214, the ARhardware system 210, and/or any other suitable system to initiatetraining using the data in the accumulated training data 222.

In some embodiments, the computing device 100, the AR device 120, thesystem 200, or a combination thereof may perform the methods describedherein. However, the methods described herein as performed by thecomputing device 100, the AR device 120, the system 200, or acombination thereof are not meant to be limiting, and any type ofsoftware executed on a computing device or a combination of variouscomputing devices can perform the methods described herein withoutdeparting from the scope of this disclosure.

FIG. 5 is a flow diagram generally illustrating a pharmacy order fillmethod 500 according to the principles of the present disclosure. At502, the method 500 begins. At 504, the method 500 determines whether toenter a training mode. For example, the computing device 100 may receiveinput from a user or programmer of the computing device 100. The inputmay indicate instructions to enter into a training mode or instructionsto enter into a non-training mode, such as a production mode, use mode,object detection mode, or other suitable mode of operation. If thecomputing device 100 determines that the input indicates instructions toenter into a training mode, the method 500 continues at 506. If thecomputing device 100 determines that the input does not indicateinstructions to enter into a training mode, the method 500 continues at534.

At 506, the method 500 connects to an order interface. For example, theRPA system 202 may connect to order website 204 or other suitableinterface to obtain order information for a pharmacy order. At 508, themethod 500 determines current workstation inventory. For example, thebox fill detection system 214 may access workstation inventory datastored in the process data 208. Additionally, or alternatively, thecurrent workstation inventory may be manually supplied by the user. Thebox fill detection system 214 may determine a current workstationinventory of the workstation 402 based on the workstation inventorydata.

In some embodiments, the box fill detection system 214 may use theartificial intelligence 216 to determine a current workstationinventory. For example, the box fill detection system 214 may access orreceive image data from the AR device 120. The box fill detection system214, using the artificial intelligence 216, may analyze the image data,as described, to identify objects in image frames of the image data. Thebox fill detection system 214, using the artificial intelligence 216,may determine whether objects detected in the image frames areworkstation inventory items or other items on or near the workstation402 based on the classification of the objects detected in the imageframes. The box fill detection system 214 may update the workstationinventory data stored in the process data 208 based on the detectedworkstation inventory items.

In some embodiments, the box fill detection system 214 may identifyinventory retrieved from the warehouse. The box fill detection system214 my increment a count associated with items on a workstationinventory list based on the retrieved inventory. For example, the boxfill detection system 214 may identify a first item retrieved from thewarehouse. The box fill detection system 214 determines whether thefirst item is on the workstation inventory list. If the box filldetection system 214 determines the first item is on the workstationinventory list, the box fill detection stem 214 increments the number offirst items on the workstation inventory list by the number of firstitems retrieved form the warehouse. If the box fill detection system 214determines that the first item is not on the workstation inventory list,the box fill detection system 214 creates an entry on the workstationinventory list corresponding to the first item and indicates the numberof first items retrieved from the warehouse.

In some embodiments, the box fill detection system 214 may be configuredto update the workstation inventory list in response to inventory itemsbeing placed in the container 406 or taken out of the container 406. Forexample, the box fill detection system 214 may decrement a countassociated with an item on the workstation inventory list in response tothe box fill detection system 214 determining that the item passed thecontainer threshold 408 and was placed in the container 406. Conversely,the box fill detection system 214 may increment a count associated withan item on the workstation inventory list in response to the box filldetection system 214 determining that the item passed the containerthreshold 408 and was removed from the container 406.

At 510, the method 500 determines pharmacy order fill requirements. Forexample, box fill detection system 214 may access intermediate orderfill data stored on the process data 208 in order to determine pharmacyorder fill requirements. The pharmacy order fill requirements mayinclude a type and quantity (e.g. or amount) of medicine, a type andquantity (e.g., or amount) of accessories, pharmacy user information(e.g., data of birth, insurance information, delivery information, othersuitable information or a combination thereof), special instructions,other suitable requirements, or a combination thereof.

At 512, the method 500 determines whether to restock the workstationinventory. For example, the box fill detection system 214 may comparingthe determined workstation inventory with the pharmacy order fillrequirements. If the box fill detection system 214 determines that theworkstation inventory includes all items in the pharmacy order fillrequirements, the method 500 continues at 520. If the box fill detectionsystem 214 determines that the workstation inventory does not includeall of the items in the pharmacy order fill requirements of if the boxfill detection system 214 determines that the workstation inventory isbelow a threshold of inventory items, the method 500 continues at 514.

At 514, the method 500 identifies a timestamp at the beginning of atraining video. For example, the box fill detection system 214 maycommunicate with the AR device 120. The box fill detection system 214may selectively instruct the AR device 120 to begin capturing data, suchas video data comprising a plurality of images, while the workstationinventory is being restocked. The AR device 120 may begin capturingvideo. The box fill detection system 214 may overlay or insert atimestamp at the beginning of the training video to indicate the starttime of the training video. At 516, the method 500 restocks theworkstation. For example, the user of the AR device 120 may retrieveitems from the stockroom, as described. The AR device 120 captures theimage data while the user of the AR device 120 retrieves the items fromthe stockroom and restocks the workstation 402. The box fill detectionsystem 214, using the artificial intelligence 216, may determine, basedon image data of the workstation 402, that the workstation inventory isrestocked. At 518, the method 500 identifies a timestamp at the end ofthe training video and stores the training video. For example, the boxfill detection system 214 instructs the AR device 120 to stop capturingimage data. The box fill detection system 214 overlays or inserts atimestamp at the end of the training video to indicate the end time ofthe training video. The box fill detection system 214 stores thetraining video in the accumulated training data 222.

At 520, the method 500 connects to an order validation page. Forexample, RPA system 202 may connect to the order website 204. At 522,the method 500 determines expected order values from the validationpage. For example, the RPA system 202 may retrieve expected order datafrom the external order information 206 via the order website 204.

At 524, the method 500 identifies a timestamp at the beginning of atraining video. For example, the box fill detection system 214 maycommunicate with the AR device 120. The box fill detection system 214may selectively instruct the AR device 120 to begin capturing data, suchas video data comprising a plurality of images, while the order is beingfilled by the user of the AR device 120. The AR device 120 may begincapturing video. The box fill detection system 214 may overlay or inserta timestamp at the beginning of the training video to indicate the starttime of the training video. At 526, the method 500 fills the order. Forexample, the user of the AR device 120 may fill the pharmacy order, asdescribed. The AR device 120 captures the image data while the user ofthe AR device 120 places items in the container 406. The box filldetection system 214, using the artificial intelligence 216, maydetermine, based on image data of the workstation 402 and the identifiedcontainer threshold 408, that the container 406 includes all requireditems of the pharmacy order.

At 528, the method 500 submits order information. For example, the boxfill detection system 214 may generate order information. The orderinformation may indicate that the pharmacy order is filled.Additionally, or alternatively, the order information may indicateshipping and/or tracking information of the pharmacy order. The box filldetection system 214 may communicate the order information to theprocess data 208. The RPA system 202 may retrieve the order informationfrom the process data 208. The RPA system 202 may communicate the orderinformation to the external order information 206 via the order website204.

At 530, the method 500 identifies a timestamp at the end of the trainingvideo and saves the training video. For example, the box fill detectionsystem 214 instructs the AR device 120 to stop capturing image data. Thebox fill detection system 214 overlays or inserts a timestamp at the endof the training video to indicate the end time of the training video.The box fill detection system 214 stores the training video in theaccumulated training data 222. At 532, the method 500 determined whetherthere are other orders to fill. For example, the RPA system 202 mayaccess the external order information 206 via the order web site 204 todetermine whether there are other pharmacy orders to fill. If RPA system202 determines that they are other pharmacy orders to fill, the method500 continues at 510. If the RPA system 202 determines that there are nopharmacy orders to fill, the method 500 ends at 560.

At 534, the method 500 connects to an order interface. For example, theRPA system 202 may connect to order website 204 or other suitableinterface to obtain order information for a pharmacy order. At 536, themethod 500 determines current workstation inventory. For example, thebox fill detection system 214 may access workstation inventory datastored in the process data 208. The box fill detection system 214 maydetermine a current workstation inventory of the workstation 402 basedon the workstation inventory data.

In some embodiments, the box fill detection system 214 may use theartificial intelligence 216 to determine a current workstationinventory. For example, the box fill detection system 214 may access orreceive image data from the AR device 120. The box fill detection system214, using the artificial intelligence 216, may analyze the image data,as described, to identify objects in image frames of the image data. Thebox fill detection system 214, using the artificial intelligence 216,may determine whether objects detected in the image frames areworkstation inventory items or other items on or near the workstation402 based on the classification of the objects detected in the imageframes. The box fill detection system 214 may update the workstationinventory data stored in the process data 208 based on the detectedworkstation inventory items.

In some embodiments, the box fill detection system 214 may identifyinventory retrieved from the warehouse. The box fill detection system214 my increment a count associated with items on a workstationinventory list based on the retrieved inventory. For example, the boxfill detection system 214 may identify a first item retrieved from thewarehouse. The box fill detection system 214 determines whether thefirst item is on the workstation inventory list. If the box filldetection system 214 determines the first item is on the workstationinventory list, the box fill detection stem 214 increments the number offirst items on the workstation inventory list by the number of firstitems retrieved form the warehouse. If the box fill detection system 214determines that the first item is not on the workstation inventory list,the box fill detection system 214 creates an entry on the workstationinventory list corresponding to the first item and indicates the numberof first items retrieved from the warehouse.

In some embodiments, the box fill detection system 214 may be configuredto update the workstation inventory list in response to inventory itemsbeing placed in the container 406 or taken out of the container 406. Forexample, the box fill detection system 214 may decrement a countassociated with an item on the workstation inventory list in response tothe box fill detection system 214 determining that the item passed thecontainer threshold 408 and was placed in the container 406. Conversely,the box fill detection system 214 may increment a count associated withan item on the workstation inventory list in response to the box filldetection system 214 determining that the item passed the containerthreshold 408 and was removed from the container 406.

At 538, the method 500 determines pharmacy order fill requirements. Forexample, box fill detection system 214 may access intermediate orderfill data stored on the process data 208 in order to determine pharmacyorder fill requirements. The pharmacy order fill requirements mayinclude a type and quantity (e.g. or amount) of medicine, a type andquantity (e.g., or amount) of accessories, pharmacy user information(e.g., data of birth, insurance information, delivery information, othersuitable information or a combination thereof), special instructions,other suitable requirements, or a combination thereof. At 540, themethod 500 determines whether to restock the workstation inventory. Forexample, the box fill detection system 214 may comparing the determinedworkstation inventory with the pharmacy order fill requirements. If thebox fill detection system 214 determines that the workstation inventoryincludes all items in the pharmacy order fill requirements, the method500 continues at 544. If the box fill detection system 214 determinesthat the workstation inventory does not include all of the items in thepharmacy order fill requirements of if the box fill detection system 214determines that the workstation inventory is below a threshold ofinventory items, the method 500 continues at 542.

At 542, the method 500 restocks the workstation. For example, the userof the AR device 120 may retrieve items from the stockroom, asdescribed. The AR device 120 captures image data, such as video datacomprising a plurality of images, while the user of the AR device 120retrieves the items from the stockroom and restocks the workstation 402.The box fill detection system 214, using the artificial intelligence216, may determine, based on image data of the workstation 402, that theworkstation inventory is restocked.

At 544, the method 500 connects to the order validation page. Forexample, RPA system 202 may connect to the order website 204. At 546,the method 500 determines expected order values from the validationpage. For example, the RPA system 202 may retrieve expected order datafrom the external order information 206 via the order website 204.

At 548, the method 500 fills the order. For example, the user of the ARdevice 120 may fill the pharmacy order, as described. The AR device 120captures the image data while the user of the AR device 120 places itemsin the container 406. The box fill detection system 214, using theartificial intelligence 216, may determine, based on image data of theworkstation 402 and the identified container threshold 408, that thecontainer 406 includes all required items of the pharmacy order. At 550,the method 500 validates on-screen order qualities and submits orderdata. For example, the box fill detection system 214 may validate thatinformation displayed on the AR device 120, the display 106, and/or theorder website 204 is accurate. The box fill detection system 214 maycompare expected quantities or values with the information displayed.The box fill detection system 214 may then generate the orderinformation (e.g., or order data) and may communicate the orderinformation, via the order website 204, to the external orderinformation 206.

At 552, the method 500 saves the order fill video. For example, the boxfill detection system 214 instructs the AR device 120 to stop capturingimage data. The box fill detection system 214 stores the image datacaptured while the pharmacy order was filled in the process data 208. At554, the method 500 determines whether overrides or errors occurringwhile filling the pharmacy order. The user of the AR device 120 mayoverride an output, such as a decision, a detection, a classification,an indication, or other output of the box fill detection system 214. Forexample, the user of the AR device 120 may view information provided bythe box fill detection system 214 on one or more lenses of the AR device120. The information may indicate that, for example, the pharmacy orderhas been filled (e.g., all required items are in the container 406). Theuser of the AR device 120 may determine that the box fill detectionsystem 214 mistakenly determined that the container 406 included allrequired items of the pharmacy order and may continue to fill thepharmacy order. The user of the AR device 120 may indicate the override,using any suitable input, to the box fill detection system 214.

If the box fill detection system 214 determines that no override orerrors occurred, the method 500 continues at 556. If the box filldetection system 214 determines that at least one override or erroroccurred, then the method 500 continues at 558. At 556, the method 500determined whether there are other orders to fill. For example, the RPAsystem 202 may access the external order information 206 via the orderweb site 204 to determine whether there are other pharmacy orders tofill. If RPA system 202 determines that they are other pharmacy ordersto fill, the method 500 continues at 538. If the RPA system 202determines that there are no pharmacy orders to fill, the method 500ends at 560. At 558, the method 500 saves the order fill video andexpected order data as training data. For example, the box filldetection system 214 stores the image data and the expected order filldata in the accumulated training data 222.

FIG. 6 is a flow diagram generally illustrating an initial objectdetection model training method 600 according to the principles of thepresent disclosure. At 602, the method 600 begins. At 604, the method600 retrieves training data. For example, the box fill detection system214 may retrieve training data including inventory-fill video frames andinventory-fill data 620; and order-fill video frames and expected orderdata 622. The data 620 and the data 622 may be stored in the accumulatedtraining data 222 or provided to the box detection system 214 by aprogrammer or user of the AR device 120 and/or the box fill detectionsystem 214. The data 620 and 622 may include image data, such videoscomprising a plurality of images.

At 606, the method 600 segments images with bounding boxes and classlabels. For example, the programmer or user of the AR device 120 and/orthe box fill detection system 214 may manually segment the plurality ofimages using bounding box information and class label information. Thebounding box information may include a point, width, and heightindicating the location of an object of interest in an image. Forexample, the bounding box information may include a box around theobject of interest in the image. The class label information indicatesthe class of the object of interest. For example, the class labelinformation may indicate that a general class of the object of interestis a container and/or the class label information may indicate that aspecific class of the object of interest is a container for a particulartype of medicine.

At 608, the method 600 divides the images into data sets for training,testing, and/or validation. For example, the programmer or user of theAR device 120 and/or the box fill detection system 214 may generate atraining set of images comprising at least some of the images of theplurality of images. The training set of images may be provided, as willbe described, to the artificial intelligence 216. The artificialintelligence 216 analyzes the objects of interest in the training set ofimages, including the bounding box information and the class labelinformation for each of the corresponding images and stores informationthat the indicates how to detect similar objects of interest in otherimages. The artificial intelligence 216 may generate and store a modelcorresponding to training set of images.

The programmer or user of the AR device 120 and/or the box filldetection system 214 may generate a testing set of images comprising atleast some of the images of the plurality of images. The testing set ofimages may be provided to the artificial intelligence 216. Theartificial intelligence 216 may load a model corresponding to thetesting set of images. The artificial intelligence 216 may use the modelto identify objects of interest in the testing set of images.

The programmer or user of the AR device 120 and/or the box filldetection system 214 may generate a validation set of images comprisingat least some of the images of the plurality of images. The validationset of images may be provided to the artificial intelligence 216. Themodels of the artificial intelligence 216 may be validated using thevalidation set of images.

At 610, the method 600 may preprocess the images. For example, theartificial intelligence 216 may preprocess the images provided to theartificial intelligence. The artificial intelligence 216 may beconfigured to preprocess the images by transforming the colors of theimages, by changing the data structure of the images (e.g., from a jpgfile type to a NumPy array), by changing dimensions of one or more ofthe images, by performing any other suitable preprocessing function, ora combination thereof. At 612, the method 600 determines a trainingconfiguration. For example, the artificial intelligence 216 may receiveinput from the programmer or user of the AR device 120 and/or the boxfill detection system 214 indicating the training configuration.

At 614, the method 600 executes the training using the configuration.For example, the artificial intelligence 216 analyzes the objects ofinterest in the training set of images, including the bounding boxinformation and the class label information for each of thecorresponding images and stores a corresponding model that the indicateshow to detect similar objects of interest in other images.

At 616, the method 600 evaluates the trained model on the test dataset.For example, the programmer or user of the AR device 120 and/or the boxfill detection system 214 may provide the artificial intelligence 216with the testing set of images. The artificial intelligence 216 may loada model corresponding to the testing set of images. The artificialintelligence 216 may use the model to identify objects of interest inthe testing set of images. The artificial intelligence 216 outputsinformation indicating what the artificial intelligence 216 predicts theobjects of interest in the testing set of images are.

The output may include bounding box information and class labelinformation. The bounding box information may include a point, width,and height indicating the location of the objects of interest in thetesting images. The class label information indicates the class of theobject of interest. The class may be generic or specific. For example,the model may be trained to identify a generic class, such as acontainer, or a specific class, such as a container for a particularmedicine. In some embodiments, the model may be trained to identify ageneric class, such as a container and the box fill detection system 214may be configured to read a barcode of the container, as described, tospecifically classify the objects of interest.

At 618, the method 600 determines whether the results of the model testare satisfactory. For example, as described, the programmer or user ofthe AR device 120 and/or the box fill detection system 214 may use thevalidation set of images to validate the models of the artificialintelligence 216. Additionally, or alternatively, the programmer or userof the AR device 120 and/or the box fill detection system 214 maymonitor a number of errors in the predictions provided by the artificialintelligence 216 during the test of the model.

If the results are satisfactory (e.g., the model validation wassuccessful or the number of errors were below a threshold), the method600 ends at 624. If the results are not satisfactory (e.g., the modelvalidation was unsuccessful or the number of errors were above thethreshold), the method 600 continues at 612. In some embodiments, themodels generated by the artificial intelligence 216 may be trainedand/or retrained using additional training sets, testing sets, andvalidation sets. The additional training sets, testing sets, andvalidation sets may be generated using any suitable images including butnot limited to, the videos captured by the AR device 120 during apharmacy order fulfillment and/or during a workstation restocking.

FIG. 7 is a flow diagram generally illustrating an object detectionmethod 700 according to the principles of the present disclosure. At702, the method 700 begins. At 704, the method 700 loads a model. Forexample, the artificial intelligence 216 loads a model trained todetected pharmacy items. At 706, the method 700 reads a frame from avideo feed. For example, the artificial intelligence 216 analyzes animage frame from video data comprising a plurality of images of theworkstation 402. At 708, the method 700 preprocesses the image frame.For example, the artificial intelligence 216 preprocesses the imageframe (e.g., using coloring, data structure conversion, and the like).

At 710, the method 700 detects an object. For example, the artificialintelligence 216 detects at least one object of interest in the imageframe. At 712, the method 700 classifies the detected object. Forexample, the artificial intelligence 216 classified the detected objectof interest, as described. At 712, the method 700 identifies a containerthreshold. For example, the artificial intelligence 216 may identify acontainer, such as the container 406. The artificial intelligence 216may identify a threshold, such as the threshold 408, of the container406, as described. For example, the artificial intelligence 216 may usecolor filtering, edge detection (e.g., Canny edge detection or othersuitable edge detection), line detection (e.g., Hough transformationline detection or other suitable line detection), or a combinationthereof to identify the threshold 408. For example, the artificialintelligence 216 may apply a Gaussian filter to smooth the image toremove the noise from the digital image, find the intensity gradients inthe image, apply non-maximum suppression to reduce spurious response toedge detection, apply double threshold to determine potential edges inthe image, track the edge(s) by hysteresis, and finalize the detectionof edge(s) by suppressing all the other edges that are weak and notconnected to strong edges.

At 716, the method 700 counts objects using the detected threshold. Forexample, the artificial intelligence 216 continues to analyze images ofthe video data provided by the AR device 120 of the workstation 402. Theartificial intelligence 216 counts classified objects as they pass thethreshold 408 and are placed, by the user of the AR device 120, into thecontainer 406. At 718, the method 700 updates an objects in containerlist. For example, the box fill detection system 214 may updated theobjects in container list to indicate the objects identified by theartificial intelligence 216 as having passed the threshold 408 into thecontainer 406. At 720, the method 700 updates a workstation inventory.For example, the box fill detection system 214 updates a workstationinventory list based on the classified objects in the container 406(e.g., and no longer in inventory).

At 724, the method 700 updates items in an order count. For example, theRPA system 202 and/or the box fill detection system 214, may update anitems in order count displayed on the lenses of the AR device 120 and/oron the order website 204 indicating the classified objects in thecontainer 406. At 726, the method 700 superimposes threshold anddetected boxes and any other labels on the image frame and the AR device120. For example, the box fill detection system 214 may superimpose thethreshold 408 on the container 406, the classified objects, and variousother labels on the image frame and/or one or more lenses of the ARdevice 120, as described.

At 728, the method 700 writes the image frame to output video. Forexample, the box fill detection system 214 may write the image frame,including the superimposed threshold 408, classified objects, andvarious other labels, to an output video. The output video may becommunicated to the order website 204. A pharmacy user may access theoutput video in order to view fulfillment progress of the pharmacyorder. Additionally, or alternatively, the output video may be stored onthe accumulated training data 222, as described.

At 730, the method 700 determines whether the items in the container isthe same as the expected items in the container. For example, the boxfill detection system 214 may compare the items (e.g., classifiedobjects) in the container 406 to the order requirements, as described.If the box fill detection system 214 determines that the items in thecontainer 406 are the same as the expected items indicated by the orderrequirements, the method 700 ends at 732. If the box fill detectionsystem 214 determines that the items in the container 406 are not thesame as the expected items indicated by the order requirements, themethod 700 continues at 706.

FIG. 8 is a flow diagram generally illustrating an object classificationdetection method 800 according to the principles of the presentdisclosure. At 802, the method 800 begins. At 804, the method 800determines whether objects are in the image frame. For example, theartificial intelligence 216 determines whether any objects of interestare in the image frame. If the artificial intelligence 216 determinesthat there are no objects of interest in the image frame, the method 800ends at 834. If the artificial intelligence 216 determines that at leastone object of interest is in the image frame, the method 800 continuesat 806.

At 806, the method 800 gets the next detected object in the image frame.For example, the artificial intelligence 216 detects the objects ofinterest in the image frame, as described. The box fill detection system214 retrieves or receives the detected object of interest. At 808, themethod 800 determines whether the object is fully classified. Forexample, the box fill detection system 214 determines whether theartificial intelligence 216 fully classified the object of interest(e.g., specifically classified the object of interest, as described). Ifthe box fill detection system 214 determines that the objects ofinterest is fully classified, the method 800 continues at 810. If thebox fill detection system 214 determines that the object of interest isnot fully classified (e.g., the artificial intelligence 216 genericallyclassified the object of interest, as described), the method 800continues at 824.

At 824, the method 800 determines whether the object type is expected.For example, the box fill detection system 214 determines whether thegeneric classification of the object of interest includes a type ofobject that is included in the items corresponding to the orderrequirements of the pharmacy order. If the box fill detection system 214determines that the type of object is not expected, the method 800continues at 820. If the box fill detection system 214 determines thatthe type of object is expected, the method 800 continues at 826. At 826,the method 800 defines a barcode search area within the object boundingbox. For example, the box fill detection system 214 generates thebarcode reader box 420, as described. At 828, the method 800 passes asearch area image into the barcode decoder. For example, the box filldetection system 214 may use a barcode decoder mechanism to read abarcode on the objects of interest in the barcode reader box 420.

At 830, the method 800 determines whether the barcode was successfullyread. For example, the box fill detection system 214 determines whetherthe barcode decoder successfully read (e.g., decoded) the barcode. Ifthe box fill detection system 214 determines that the barcode decodesuccessfully read the barcode, the method 800 continues at 810. If thebox fill detection system 214 determines that the barcode decoder didnot successfully read the barcode, the method 800 continues at 832. At832, the method 800 prompts the user of the AR device 120 to rotate theobject. For example, the box fill detection system 214 may provide theindication 414, as described instructing the user of the AR device 120to reposition the object of interest, such that the barcode is withinthe barcode reader box 420.

At 810, the method 800 determines whether the object is expected. Forexample, the box fill detection system 214 determines whether the object(e.g. being fully classified) is an expected item according to the orderdata corresponding to the pharmacy order. If the box fill detectionsystem 214 determines that the object of interest is an expected object,the method 800 continues at 812. If the box fill detection system 214determine that the object of interest is not an expected object, themethod 800 continues at 820. At 820, the method 800 determines whetherthe classified object corresponds to fulfillment of a pharmacy order.For example, the box fill detection system 214 determine whether theobject of interest corresponds to fulfillment of the pharmacy order orto a workstation restocking. If the box fill detection system 214determines that the object of interest corresponds to fulfillment of thepharmacy order, the method 800 continues at 822. If the box filldetection system 214 determines that the object of interest correspondsto a workstation restocking, the method 800 continues at 806.

At 822, the method 800 generates an override. For example, the box filldetection system 214 may generate output indicating that the objects ofinterest was not expected. At 812, the method 800 determines whether alabel is needed for the object of interest. For example, the box filldetection system 214 may determine whether a label is required for theobject of interest. The box fill detection system 214 may useinformation indicated in the order requirements to determine whether theobject of interest requires a label. If the box fill detection system214 determines that a label is not required, the method 800 continues at806. If the box fill detection system 214 determines that a label isrequired, the method 800 continues at 814.

At 814, the method 800 determines if a label is already printed. Forexample, the box fill detection system 214 determines whether a labelhas been printed. For example, the box fill detection system 214 mayreceive an indication that the label is already printed or the box filldetection system 214, using the artificial intelligence 216, mayidentify a label in the image frames. If the box fill detection system214 determines that the label has not been printed, the method 800continues at 818. If the box fill detection system 214 determines that alabel has already been printed, the method 800 continues at 816. At 818,the method 800 prints a label. The method 800 continues at 806. At 816,the method 800 scans the label if capable of being scanned. For example,the box fill detection system 214, using the artificial intelligence 216or a scanning device, scans the label. The method 800 continues at 806.

FIG. 9 is a flow diagram generally illustrating a workstation restockingmethod 900 according to the principles of the present disclosure. At902, the method 900 begins. At 904, the method 900 loads a model. Forexample, the artificial intelligence 216 may load a model trained torestock a workstation. At 906, the method 900 loads guidance torecommended item location. For example, the AR hardware system 210 mayretrieve information from the warehouse data records 212 includingwarehouse layout information, warehouse inventory information, inventorylocation information, other suitable information, or a combinationthereof. At 908, the method 900 reads an image frame of the video feed.For example, the AR hardware system 210 reads an image frame of thevideo feed of the workstation 402, as described.

At 910, the method 900 preprocesses the image frame. For example, theartificial intelligence 216 preprocesses the image frame, as described.At 912, the method 900 detects objects. For example, the artificialintelligence 216 detects objects of interest in the image frame, asdescribed. At 914, the method 900 classifies the detected object. Forexample, the artificial intelligence 216 classifies the detected objectas described.

At 916, the method 900 determines whether the user of the AR device 120made a gestural indication. For example, the user of the AR device 120may perform a gesture to indicate that an inventory container or itemhas been retrieved. The gesture may include one or more motions toretrieve the inventory container or item, a specific gesture, such as apredetermined hand movement, or other suitable gesture. The AR hardwaresystem 210 and/or the artificial intelligence 216 may be configured torecognize the gesture and may provide a visual acknowledgement, such asthe visual acknowledgement 308, that the user has retrieved theinventory container or item. If the AR hardware system 210 or theartificial intelligence 216 determines that the user of the AR device120 made a gestural indication, the method 900 continues at 918. If theAR hardware system 210 or the artificial intelligence 216 determinesthat the user of the AR device 120 did not make a gestural indication,the method 900 continues at 920.

At 918, the method 900 updates the warehouse and workstationinventories. For example, the AR hardware system 210 may update thewarehouse inventory list and/or workstation inventory list, asdescribed. At 920, the method 900 presents guidance arrows and inventoryprompts to the AR device 120. For example, the AR hardware system 210may provide the guided access to one or more lenses of the AR device120, as described. At 922, the method 900 writes the image frame to atraining video. For example, the AR hardware system 210 may write theimage frame to a training video. The AR hardware system 210 may storethe training video in the accumulated training data 222, as described.

At 924, the method 900 determines whether all items have been gathered.For example, the AR hardware system 210 may compare the items on theworkstation inventory list with the items required to fulfill thepharmacy order indicated by the order requirements. If the AR hardwaresystem 210 determines that the items on the workstation inventory listare the same as the items required to fulfill the pharmacy order, themethod 900 continues at 926. If the AR hardware system 210 determinesthat the items on the workstation inventory list are not the same as theitems required to fulfill the pharmacy order, the method 900 continue at906.

At 926, the method 900 determines whether to enter an order fill mode.For example, the AR hardware system 210 determine whether to enter anorder fill mode base on input from the user of the AR device 120 orbased on the items on the workstation inventory list being the same asthe items required to fulfill the pharmacy order. If the AR hardwaresystem 210 determines not to enter the order fill mode, the method 900continues at 906. If the AR hardware system 210 determines to enter theorder fill mode, the method ends at 928.

FIG. 10 is a flow diagram generally illustrating an alternative pharmacyorder fill method 1000 according to the principles of the presentdisclosure. At 1002, the method 1000 captures images of the pharmacyworkstation. For example, the AR device 120 captures video data includea plurality of images of the workstation 402. At 1004, the method 1000detects objects of interest in the plurality of images. For example, theartificial intelligence 216 detects objects of interest in the pluralityof images, as described. At 1006, the method 1000 classifies thedetected objects of interest. For example, the artificial intelligence216 classifies the detected objects of interest. At 1008, the method1000 retrieves a workstation inventory list. For example, the box filldetection system 214 and/or the AR hardware system 210 retrieves theworkstation inventory list, as described.

At 1010, the method 1000 identified a container boundary. For example,the box fill detection system 214, using the artificial intelligence216, detects the threshold 408 of the container 406. At 1012, the method1000 updates an objects in container list. For example, the artificialintelligence 216 identifies objects passing the threshold 408, asdescribed. The box fill detection system 214 updates the objects incontainer list based on the objects identified by the artificialintelligence 216 having passed the threshold.

In some embodiments, a method for fulfilling a pharmacy order includescapturing, by an image-capturing device, a one or more images of atleast a portion of a pharmacy workstation. The method also includesidentifying, by a processor in communication with the image capturingdevice, objects of interest in a first image of the one or more imagesand classifying, by the processor, the detected objects of interestusing a convolutional neural network associated with the processor. Themethod also includes updating, by the processor, a workstation inventorylist based on the classification of the detected objects of interest andidentifying, by the processor, a boundary defining an opening of acontainer in a second image of the one or more images. The method alsoincludes updating, by the processor, an objects in container list basedon a determination that at least one of the classified objects passedthe boundary.

In some embodiments, identifying, by the processor, the boundarydefining the opening of the container includes using color filtering. Insome embodiments, identifying, by the processor, the boundary definingthe opening of the container includes using Canny edge detection. Insome embodiments, identifying, by the processor, the boundary definingthe opening of the container includes using Hough transform linedetection. In some embodiments, classifying, by the processor, thedetected objects of interest using the convolutional neural networkassociated with the processor includes identifying barcodes on at leastsome of the objects using the convolutional neural network. In someembodiments, the method also includes providing by the processor to adisplay device, an indication corresponding to the objects in containerlist. In some embodiments, the method also includes determining, by theprocessor, whether the objects in container list is the same as anexpected objects list. In some embodiments, the method also includesindicating, by the processor on a display device, that the pharmacyorder is fulfilled based on a determination that objects in containerlist is the same as the expected objects list.

In some embodiments, an augmented reality apparatus for fulfilling apharmacy order includes an image-capturing device, a processor, and adisplay device. The image-capturing device is configured to capturevideo data that includes a one or more images of at least a portion of apharmacy workstation. The processor in communication with at least onememory that includes instructions that, when executed by the processor,cause the processor to: read a first image of the one or more images ofthe video data; identify objects of interest in the first image;classify the detected objects of interest using a convolutional neuralnetwork; identify a boundary defining an opening of a container in asecond image of the one or more images; update an objects in containerlist based on a determination that at least one of the classifiedobjects passed the boundary; and generate information corresponding tothe pharmacy order based on, at least, the objects in container list.The display device is configured to display the information.

In some embodiments, the instructions further cause the processor toidentify the boundary defining the opening of the container includesusing color filtering. In some embodiments, the instructions furthercause the processor to identify the boundary defining the opening of thecontainer includes using Canny edge detection. In some embodiments, theinstructions further cause the processor to identify the boundarydefining the opening of the container includes using Hough transformline detection. In some embodiments, the instructions further cause theprocessor to classifies the detected objects of interest by at leastidentifying barcodes on at least some of the objects using theconvolutional neural network. In some embodiments, the instructionsfurther cause the processor to update the workstation inventory listbased on the determination that at least one of the classified objectspassed the boundary. In some embodiments, the information includes anindication corresponding objects in the objects in container list. Insome embodiments, the instructions further cause the processor todetermine whether the objects in container list is the same as anexpected objects list. In some embodiments, the information includes anindication that the pharmacy order is fulfilled based on a determinationthat objects in container list is the same as the expected objects list.

In some embodiments, a system for fulfilling a pharmacy order includes awearable augmented reality device and a convolutional neural network.The wearable augmented reality device is configured to: capture videodata that includes a one or more images of at least a portion of apharmacy workstation; identify objects of interest in the one or moreimages; and identify a boundary defining an opening of a container inthe one or more images. The convolutional neural network is configuredto classify the objects of interest identified by the wearable augmentedreality device, wherein the wearable augmented reality deviceidentifies, in the one or more images, classified objects of interestthat pass the boundary of the container and provides, to a wearer of thewearable augmented reality device, information corresponding to thepharmacy order in response to identifying classified objects of interestthat pass the boundary of the container.

In some embodiments, the wearable augmented reality device is furtherconfigured to determine whether the pharmacy order is fulfilled inresponse to identifying classified objects that pass the boundary of thecontainer, wherein the information includes an indication that thepharmacy order is fulfilled.

The above discussion is meant to be illustrative of the principles andvarious embodiments of the present invention. Numerous variations andmodifications will become apparent to those skilled in the art once theabove disclosure is fully appreciated. It is intended that the followingclaims be interpreted to embrace all such variations and modifications.

The word “example” is used herein to mean serving as an example,instance, or illustration. Any aspect or design described herein as“example” is not necessarily to be construed as preferred oradvantageous over other aspects or designs. Rather, use of the word“example” is intended to present concepts in a concrete fashion. As usedin this application, the term “or” is intended to mean an inclusive “or”rather than an exclusive “or.” That is, unless specified otherwise, orclear from context, “X includes A or B” is intended to mean any of thenatural inclusive permutations. That is, if X includes A; X includes B;or X includes both A and B, then “X includes A or B” is satisfied underany of the foregoing instances. In addition, the articles “a” and “an”as used in this application and the appended claims should generally beconstrued to mean “one or more” unless specified otherwise or clear fromcontext to be directed to a singular form. Moreover, use of the term “animplementation” or “one implementation” throughout is not intended tomean the same embodiment or implementation unless described as such.

Implementations the systems, algorithms, methods, instructions, etc.,described herein can be realized in hardware, software, or anycombination thereof. The hardware can include, for example, computers,intellectual property (IP) cores, application-specific integratedcircuits (ASICs), programmable logic arrays, optical processors,programmable logic controllers, microcode, microcontrollers, servers,microprocessors, digital signal processors, or any other suitablecircuit. In the claims, the term “processor” should be understood asencompassing any of the foregoing hardware, either singly or incombination. The terms “signal” and “data” are used interchangeably.

As used herein, the term module can include a packaged functionalhardware unit designed for use with other components, a set ofinstructions executable by a controller (e.g., a processor executingsoftware or firmware), processing circuitry configured to perform aparticular function, and a self-contained hardware or software componentthat interfaces with a larger system. For example, a module can includean application specific integrated circuit (ASIC), a Field ProgrammableGate Array (FPGA), a circuit, digital logic circuit, an analog circuit,a combination of discrete circuits, gates, and other types of hardwareor combination thereof. In other embodiments, a module can includememory that stores instructions executable by a controller to implementa feature of the module.

Further, in one aspect, for example, systems described herein can beimplemented using a general-purpose computer or general-purposeprocessor with a computer program that, when executed, carries out anyof the respective methods, algorithms, and/or instructions describedherein. In addition, or alternatively, for example, a special purposecomputer/processor can be utilized which can contain other hardware forcarrying out any of the methods, algorithms, or instructions describedherein.

Further, all or a portion of implementations of the present disclosurecan take the form of a computer program product accessible from, forexample, a computer-usable or computer-readable medium. Acomputer-usable or computer-readable medium can be any device that can,for example, tangibly contain, store, communicate, or transport theprogram for use by or in connection with any processor. The medium canbe, for example, an electronic, magnetic, optical, electromagnetic, or asemiconductor device. Other suitable mediums are also available.

The above-described embodiments, implementations, and aspects have beendescribed in order to allow easy understanding of the present inventionand do not limit the present invention. On the contrary, the inventionis intended to cover various modifications and equivalent arrangementsincluded within the scope of the appended claims, which scope is to beaccorded the broadest interpretation to encompass all such modificationsand equivalent structure as is permitted under the law.

What is claimed is:
 1. A method for fulfilling an order using augmentedreality, the method comprising: storing location of at least one item inthe order; detecting current location within a warehouse in which theitem is stored; displaying, on a wearable augmented reality device,guidance within the warehouse to the item; capturing, by animage-capturing device of a wearable augmented reality device, one ormore item images at a storage location of the item; detecting retrieval,using a processor of the wearable augmented reality device, of the itemfrom the storage location; delivering the item to an order fulfillmentlocation; scanning the item at the order fulfillment location using thewearable augmented reality device; and monitoring placement of the itemin a container for delivery using the wearable augmented reality device.2. The method of claim 1, wherein monitoring includes: identifying, bythe image-capturing device, a virtual boundary defining an opening ofthe container in a second image of the one or more images;superimposing, on a display of the wearable augmented reality device, athreshold that defines a virtual opening bound by the virtual boundaryon the container; counting detected items passing the threshold into thecontainer; updating, by the processor, an objects-in-container listbased on a determination that at least one of the items passed thethreshold; and displaying, on the display of the wearable augmentedreality device, the items in container list.
 3. The method of claim 1,wherein detecting retrieval includes calculating and displaying anindicator on a display of the augmented reality device that a correctnumber of the item has been retrieved from the storage location.
 4. Themethod of claim 1, wherein capturing one or more item images includeshighlighting the item at the storage location in the warehouse when inthe field of view of an imager of the wearable augmented reality device.5. The method of claim 4, wherein detecting retrieval of the itemincludes detecting movement of the item from outside a boundary definedby the highlighting the item.
 6. The method of claim 5, whereindetecting retrieval of the item includes detecting a gesture by thewearer of the wearable augmented reality device.
 7. The method of claim4, wherein highlighting the item at the storage location includesidentifying, by a processor of the wearable augmented reality device, avirtual boundary defining the item using color filtering.
 8. The methodof claim 1, wherein detecting retrieval includes detecting, using theaugmented reality device, gestures from a wearer associated withremoving the item from the storage location and placing item in aportable container.
 9. The method of claim 1, wherein detectingretrieval includes updating inventory of the item type in the warehouseusing retrieval from the storage location.
 10. The method of claim 1,wherein displaying guidance includes displaying a directional symbol anditem information on a heads up display visible to a wearer while thewearer is moving through the warehouse toward the item.
 11. An augmentedreality apparatus for fulfilling an order, comprising: animage-capturing device configured to capture video data that includes aone or more images within a warehouse; a processor in communication withat least one memory storing image data from the image-capturing deviceand instructions that, when executed by the processor, cause theprocessor to: detect current location within a warehouse in which theitem is stored; display, on a wearable augmented reality device,guidance within the warehouse to the item; capture, by animage-capturing device of a wearable augmented reality device, one ormore item images at a storage location of the item; detect retrieval,using a processor of the wearable augmented reality device, of the itemfrom the storage location; deliver the item to an order fulfillmentlocation; scan the item at the order fulfillment location using thewearable augmented reality device; and virtually monitor placement ofthe item in a container for delivery using the wearable augmentedreality device.
 12. The augmented reality apparatus of claim 11, whereinthe instructions further cause the processor to identify, by theimage-capturing device, a virtual boundary defining an opening of thecontainer in a second image of the one or more images; superimpose, on adisplay of the wearable augmented reality device, a threshold thatdefines a virtual opening bound by the virtual boundary on thecontainer; count detected items passing the threshold into thecontainer; update, by the processor, an objects-in-container list basedon a determination that at least one of the items passed the threshold;and display, on the display of the wearable augmented reality device,the items in container list.
 13. The augmented reality apparatus ofclaim 11, wherein the instructions further cause the processor todisplay an indicator on a display of the augmented reality device that acorrect number of the item has been retrieved from the storage location.14. The augmented reality apparatus of claim 11, wherein theinstructions further cause the processor to capture one or more itemimages includes highlighting the item at the storage location in thewarehouse when in the field of view of an imager of the wearableaugmented reality device.
 15. The augmented reality apparatus of claim14, wherein the instructions further cause the processor to detectmovement of the item from outside a boundary defined by the highlightingthe item.
 16. The augmented reality apparatus of claim 15, wherein theinstructions further cause the processor to detecting a gesture by thewearer of the wearable augmented reality device.
 17. The augmentedreality apparatus of claim 14, wherein the instructions further causethe processor to identify a virtual boundary defining the item usingcolor filtering.
 18. The augmented reality apparatus of claim 11,wherein the instructions further cause the processor to detect gesturesfrom a wearer associated with removing the item from the storagelocation and placing item in a portable container.
 19. The augmentedreality apparatus of claim 11, wherein the instructions further causethe processor to update inventory of the item type in the warehouseusing retrieval from the storage location.
 20. The augmented realityapparatus of claim 11, wherein the instructions further cause theprocessor to display a directional symbol and item information on aheads up display visible to a wearer while the wearer is moving throughthe warehouse toward the item.